{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a8321ad3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 7.63 ms (started: 2021-08-13 02:21:41 +08:00)\n"
     ]
    }
   ],
   "source": [
    "# 自动计算cell的计算时间\n",
    "%load_ext autotime\n",
    "\n",
    "%config InlineBackend.figure_format='svg' #矢量图设置，让绘图更清晰"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "359536f7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "origin\tgit@github.com:ustchope/kerastuner.git (fetch)\n",
      "origin\tgit@github.com:ustchope/kerastuner.git (push)\n",
      "[master 2a9432b] 更新 #2 Aug 13, 2021\n",
      " 1 file changed, 402 insertions(+), 2 deletions(-)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "To github.com:ustchope/kerastuner.git\n",
      "   5d10e01..2a9432b  master -> master\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 3.87 s (started: 2021-08-13 02:35:37 +08:00)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# 增加更新\n",
    "git add *.ipynb\n",
    "\n",
    "git remote -v\n",
    "\n",
    "git commit -m '更新 #2 Aug 13, 2021'\n",
    "\n",
    "git push origin master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d9a6b92a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.43 s (started: 2021-08-13 02:22:09 +08:00)\n"
     ]
    }
   ],
   "source": [
    "#设置使用的gpu\n",
    "import tensorflow as tf\n",
    "\n",
    "gpus = tf.config.list_physical_devices(\"GPU\")\n",
    "\n",
    "if gpus:\n",
    "   \n",
    "    gpu0 = gpus[0] #如果有多个GPU，仅使用第0个GPU\n",
    "    tf.config.experimental.set_memory_growth(gpu0, True) #设置GPU显存用量按需使用\n",
    "    # 或者也可以设置GPU显存为固定使用量(例如：4G)\n",
    "    #tf.config.experimental.set_virtual_device_configuration(gpu0,\n",
    "    #    [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)]) \n",
    "    tf.config.set_visible_devices([gpu0],\"GPU\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1fbc090",
   "metadata": {},
   "source": [
    "# 介绍"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16444ec0",
   "metadata": {},
   "source": [
    "keras_tuner.engine.tuner.Tuner 中的 Tuner 类可以被子类化以支持高级用途，例如：\n",
    "* 自定义训练循环（GAN、强化学习等）\n",
    "* 在模型构建功能之外添加超参数（预处理、数据增强、测试时间增强等）\n",
    "\n",
    "本教程不会介绍支持非 Keras 模型的子类化。 为此，您可以将 `keras_tuner.engine.base_tuner.BaseTuner` 类子类化（参见 `keras_tuner.tuners.sklearn.Sklearn` 示例）。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa805f50",
   "metadata": {},
   "source": [
    "# 了解搜索过程"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5efd134d",
   "metadata": {},
   "source": [
    "`Tuner.search` 可以传递任何参数。 这些参数将直接传递给 `Tuner.run_trial`，以及一个 `Trial` 对象，该对象包含有关当前试验的信息，包括超参数和试验状态。 通常，`Tuner.run_trial` 是用户在子类化 Tuner 时需要覆盖的唯一方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ba9301d",
   "metadata": {},
   "source": [
    "# 重载 run_trial"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04e43ea0",
   "metadata": {},
   "source": [
    "run_trial 有两种写法。 一种是利用 Tuner 的内置回调钩子，将目标值发送到 Oracle 并保存模型的最新状态。 这些钩子是：\n",
    "* self.on_epoch_end：必须被调用。 将结果报告给 Oracle 并保存模型。 传递给此方法的日志字典必须包含目标名称。\n",
    "* self.on_epoch_begin、self.on_batch_begin、self.on_batch_end：可选。 这些方法在 Tuner 中什么都不做，但是如果您希望子类的用户创建他们自己的子类来覆盖训练过程的这些部分，那么这些方法作为钩子提供是很有用的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39e05a3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyTuner(kt.Tuner):\n",
    "\n",
    "    def run_trial(self, trial, ...):\n",
    "        model = self.hypermodel.build(trial.hyperparameters)\n",
    "        for epoch in range(10):\n",
    "              epoch_loss = ...\n",
    "              self.on_epoch_end(trial, model, epoch, logs={'loss': epoch_loss})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a16d77f",
   "metadata": {},
   "source": [
    "或者，您可以改为直接调用用于向 Oracle 报告结果并保存模型的方法。 对于没有自然时期概念或您不想在每个时期后向 Oracle 报告结果的用例，这可以提供更大的灵活性。 这些方法是：\n",
    "* self.oracle.update_trial：向 Oracle 报告当前结果。 传递给此方法的指标字典必须包含目标名称。\n",
    "* self.save_model：保存训练好的模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed435254",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyTuner(kt.Tuner):\n",
    "\n",
    "    def run_trial(self, trial, ...):\n",
    "        model = self.hypermodel.build(trial.hyperparameters)\n",
    "        score = ...\n",
    "        self.oracle.update_trial(trial.trial_id, {'score': score})\n",
    "        self.save_model(trial.trial_id, model)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d430eb6",
   "metadata": {},
   "source": [
    "## 在预处理、评估等过程中添加超参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9b4bf08",
   "metadata": {},
   "source": [
    "新的 HyperParameters 可以在 run_trial 的任何地方定义，就像在 HyperModel 中定义 HyperParameters 一样。 这些超参数在第一次遇到时采用默认值，然后由 Oracle 调整。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "427c5183",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyTuner(kt.Tuner):\n",
    "\n",
    "    def run_trial(self, trial, ...):\n",
    "        hp = trial.hyperparameters\n",
    "        model = self.hypermodel.build(hp)\n",
    "\n",
    "        batch_size = hp.Int('batch_size', 32, 128, step=32)\n",
    "        random_flip = hp.Boolean('random_flip')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2bd9d80c",
   "metadata": {},
   "source": [
    "## 端到端示例："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b96f6926",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 446 ms (started: 2021-08-13 02:32:13 +08:00)\n"
     ]
    }
   ],
   "source": [
    "import keras_tuner as kt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def build_model(hp):\n",
    "    \"\"\"Builds a convolutional model.\"\"\"\n",
    "    inputs = tf.keras.Input(shape=(28, 28, 1))\n",
    "    x = inputs\n",
    "    for i in range(hp.Int(\"conv_layers\", 1, 3, default=3)):\n",
    "        x = tf.keras.layers.Conv2D(\n",
    "            filters=hp.Int(\"filters_\" + str(i), 4, 32, step=4, default=8),\n",
    "            kernel_size=hp.Int(\"kernel_size_\" + str(i), 3, 5),\n",
    "            activation=\"relu\",\n",
    "            padding=\"same\",\n",
    "        )(x)\n",
    "\n",
    "        if hp.Choice(\"pooling\" + str(i), [\"max\", \"avg\"]) == \"max\":\n",
    "            x = tf.keras.layers.MaxPooling2D()(x)\n",
    "        else:\n",
    "            x = tf.keras.layers.AveragePooling2D()(x)\n",
    "\n",
    "        x = tf.keras.layers.BatchNormalization()(x)\n",
    "        x = tf.keras.layers.ReLU()(x)\n",
    "\n",
    "    if hp.Choice(\"global_pooling\", [\"max\", \"avg\"]) == \"max\":\n",
    "        x = tf.keras.layers.GlobalMaxPooling2D()(x)\n",
    "    else:\n",
    "        x = tf.keras.layers.GlobalAveragePooling2D()(x)\n",
    "    outputs = tf.keras.layers.Dense(10, activation=\"softmax\")(x)\n",
    "\n",
    "    model = tf.keras.Model(inputs, outputs)\n",
    "\n",
    "    optimizer = hp.Choice(\"optimizer\", [\"adam\", \"sgd\"])\n",
    "    model.compile(\n",
    "        optimizer, loss=\"sparse_categorical_crossentropy\", metrics=[\"accuracy\"]\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "af39892d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "time: 1.98 ms (started: 2021-08-13 02:34:24 +08:00)\n"
     ]
    }
   ],
   "source": [
    "class MyTuner(kt.Tuner):\n",
    "    def run_trial(self, trial, train_ds):\n",
    "        hp = trial.hyperparameters\n",
    "\n",
    "        # Hyperparameters can be added anywhere inside `run_trial`.\n",
    "        # When the first trial is run, they will take on their default values.\n",
    "        # Afterwards, they will be tuned by the `Oracle`.\n",
    "        train_ds = train_ds.batch(hp.Int(\"batch_size\", 32, 128, step=32, default=64))\n",
    "\n",
    "        model = self.hypermodel.build(trial.hyperparameters)\n",
    "        lr = hp.Float(\"learning_rate\", 1e-4, 1e-2, sampling=\"log\", default=1e-3)\n",
    "        optimizer = tf.keras.optimizers.Adam(lr)\n",
    "        epoch_loss_metric = tf.keras.metrics.Mean()\n",
    "        loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()\n",
    "\n",
    "        # @tf.function\n",
    "        def run_train_step(data):\n",
    "            images = tf.dtypes.cast(data[0], \"float32\") / 255.0\n",
    "            labels = data[1]\n",
    "            with tf.GradientTape() as tape:\n",
    "                logits = model(images)\n",
    "                loss = loss_fn(labels, logits)\n",
    "                # Add any regularization losses.\n",
    "                if model.losses:\n",
    "                    loss += tf.math.add_n(model.losses)\n",
    "            gradients = tape.gradient(loss, model.trainable_variables)\n",
    "            optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "            epoch_loss_metric.update_state(loss)\n",
    "            return loss\n",
    "\n",
    "        # `self.on_epoch_end` reports results to the `Oracle` and saves the\n",
    "        # current state of the Model. The other hooks called here only log values\n",
    "        # for display but can also be overridden. For use cases where there is no\n",
    "        # natural concept of epoch, you do not have to call any of these hooks. In\n",
    "        # this case you should instead call `self.oracle.update_trial` and\n",
    "        # `self.oracle.save_model` manually.\n",
    "        for epoch in range(2):\n",
    "            print(\"Epoch: {}\".format(epoch))\n",
    "\n",
    "            self.on_epoch_begin(trial, model, epoch, logs={})\n",
    "            for batch, data in enumerate(train_ds):\n",
    "                self.on_batch_begin(trial, model, batch, logs={})\n",
    "                batch_loss = float(run_train_step(data))\n",
    "                self.on_batch_end(trial, model, batch, logs={\"loss\": batch_loss})\n",
    "\n",
    "                if batch % 100 == 0:\n",
    "                    loss = epoch_loss_metric.result().numpy()\n",
    "                    print(\"Batch: {}, Average Loss: {}\".format(batch, loss))\n",
    "\n",
    "            epoch_loss = epoch_loss_metric.result().numpy()\n",
    "            self.on_epoch_end(trial, model, epoch, logs={\"loss\": epoch_loss})\n",
    "            epoch_loss_metric.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "33dc26d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trial 2 Complete [00h 00m 02s]\n",
      "loss: 2.302398920059204\n",
      "\n",
      "Best loss So Far: 2.267237424850464\n",
      "Total elapsed time: 00h 00m 09s\n",
      "INFO:tensorflow:Oracle triggered exit\n",
      "time: 11.2 s (started: 2021-08-13 02:34:42 +08:00)\n"
     ]
    }
   ],
   "source": [
    "tuner = MyTuner(\n",
    "    oracle=kt.oracles.BayesianOptimization(\n",
    "        objective=kt.Objective(\"loss\", \"min\"), max_trials=2\n",
    "    ),\n",
    "    hypermodel=build_model,\n",
    "    directory=\"results\",\n",
    "    project_name=\"mnist_custom_training\",\n",
    ")\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
    "\n",
    "# Reshape the images to have the channel dimension.\n",
    "x_train = x_train.reshape(x_train.shape + (1,))[:1000]\n",
    "y_train = y_train.astype(np.int64)[:1000]\n",
    "\n",
    "mnist_train = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
    "\n",
    "tuner.search(train_ds=mnist_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "30e08102",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'conv_layers': 2, 'filters_0': 28, 'kernel_size_0': 4, 'pooling0': 'avg', 'filters_1': 24, 'kernel_size_1': 3, 'pooling1': 'avg', 'filters_2': 28, 'kernel_size_2': 4, 'pooling2': 'max', 'global_pooling': 'max', 'optimizer': 'adam', 'batch_size': 64, 'learning_rate': 0.001}\n",
      "time: 177 ms (started: 2021-08-13 02:34:58 +08:00)\n"
     ]
    }
   ],
   "source": [
    "best_hps = tuner.get_best_hyperparameters()[0]\n",
    "print(best_hps.values)\n",
    "\n",
    "best_model = tuner.get_best_models()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fb2fa942",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         [(None, 28, 28, 1)]       0         \n",
      "_________________________________________________________________\n",
      "conv2d (Conv2D)              (None, 28, 28, 28)        476       \n",
      "_________________________________________________________________\n",
      "average_pooling2d (AveragePo (None, 14, 14, 28)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 14, 14, 28)        112       \n",
      "_________________________________________________________________\n",
      "re_lu (ReLU)                 (None, 14, 14, 28)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 14, 14, 24)        6072      \n",
      "_________________________________________________________________\n",
      "average_pooling2d_1 (Average (None, 7, 7, 24)          0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 7, 7, 24)          96        \n",
      "_________________________________________________________________\n",
      "re_lu_1 (ReLU)               (None, 7, 7, 24)          0         \n",
      "_________________________________________________________________\n",
      "global_max_pooling2d (Global (None, 24)                0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                250       \n",
      "=================================================================\n",
      "Total params: 7,006\n",
      "Trainable params: 6,902\n",
      "Non-trainable params: 104\n",
      "_________________________________________________________________\n",
      "time: 10.9 ms (started: 2021-08-13 02:35:21 +08:00)\n"
     ]
    }
   ],
   "source": [
    "best_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb523521",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:tf2keras]",
   "language": "python",
   "name": "conda-env-tf2keras-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
